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Achieving the Dream colleges engage in a process of institutional improvement to 
increase student success. A central component of this process is engaging internal 
and external stakeholders to help develop and implement interventions or changes 
in programs and services that improve student success. To determine whether 
these interventions do indeed improve student outcomes, and what changes and 
refinements should be made to produce further improvements, colleges need to 
evaluate their interventions. 


Achieving the Dream institutions are expected to develop 
plans for evaluating student success interventions before 
they begin their implementation. These plans should 
specify the collection and analysis of quantitative and 
qualitative data that will form the basis of the evaluation. 
The plans should also indicate how the results of the 
evaluation will be used to improve the effectiveness of the 
interventions. 

Having the capacity to effectively evaluate student 
success interventions is necessary for meeting the 
expectations of funders, accreditation agencies, 
students, policy makers, and the public. It is essential 
for continuously improving the impact of programs and 
services on student learning and success. 

This guide is designed to help colleges plan and 
conduct effective evaluations of their student success 
interventions. 


What is Evaluation? 

We are constantly evaluating. From the moment we walk 
into our offices, whenever we decide which e-mails to 
answer immediately or even when to schedule meetings, 
we are engaged in evaluation. When faced with a decision, 
we compare the available data with some criteria, and then 
we decide. When students try to decide whether to take 
a particular course, they consider various factors, such as 
the number of credit hours, the faculty teaching it, the 
schedule, how many students have failed or succeeded in 
the class, and whether it is required or merely an elective. 
These factors are the criteria upon which they base their 
judgment. Thus, an easily remembered definition for 
evaluation is: Evaluation is a value judgment based 
on detensible criteria. 

The Program Evaluation Standards represent principles 
by which many institutions, evaluation offices, or teams 
measure their evaluation efforts. When evaluating an 
intervention, these standards are points of reference 
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used to determine, among other things, whether 
the intervention is producing the expected results, 
and whether there is room for improvement, if the 
intervention can be brought to scale, or if it should simply 
be terminated. 

It is important to distinguish between assessment and 
evaluation, and between research and evaluation. These 
terms are frequently used interchangeably. Assessment 
generally refers to the process of determining the extent 
to which students have mastered some instructional 
objective or competency. Evaluation, on the other hand, 
uses the same information and other criteria not only to 
determine the extent to which a performance measure 
has been met, but also to compare it with other criteria 
in order to make a decision. This decision could include 
offering the student some supplemental instruction, 
suggesting that the student take a different approach to 
note-taking, or asking the student to retake the course. 

In short, a key difference between assessment and 
evaluation is that the latter involves a value judgment. 

Evaluation and research also differ. Carol Weiss has 
identified 13 different ways in which evaluation and 
research differ. For example, evaluation is intended for 
use, while research produces knowledge and the natural 
process of dissemination determines its use. Evaluation 
compares what is with what should be, posing the 
question: does it meet the established criteria.? Research, 
on the other hand, simply studies what is and what should 
be. In addition, evaluation takes place in an action setting 
where the intervention, rather than the evaluation itself, 
has priority. With research, the priority is given to the 
research itself and not to practical solutions. 

Five Steps for Effectively Evaluating Student 
Success Interventions 

In this guide, evaluation is presented as a five-step 
process: 

1 . Describe the intervention 

2 . Identify the evaluation questions 

3 . Complete the evaluation plan 

4 . Monitor the execution of the plan 

5 . Learn, share, and use the results. 

The following provides details on each of the steps. 


Step 1 : Describe the intervention 

Achieving the Dream colleges should clearly describe 
the interventions being evaluated, including their 
goals and objectives, target population, and duration. 
Intervention descriptions should clearly articulate how 
the intervention will bring about the desired outcomes in 
student achievement. A useful tool for accomplishing this 
is a logic model. A logic model is a graphical illustration of 
the actions and activities involved in an intervention and 
how they are expected to achieve the stated objectives. 
Logic models help teams implementing interventions to 
understand clearly the resources and time commitment 
that a particular intervention will require and the 
outcomes it is expected to deliver. A logic model is useful 
in helping define the questions to be answered through 
the evaluation. 

Figure 1 shows the typical elements of a linear logic 
model: 

Inputs are all the resources that are used during the life 
of an intervention. These include staff, time, money, 
material, equipment, technology, volunteers, partners, etc. 

Activities are all the specific tasks executed during 
an intervention — for example, complete a literature 
review, train faculty, facilitate meetings, develop a 
new curriculum, or add new elements to the student 
orientation process. 

Outputs are the direct results or products generated by the 
completion of activities. These are sometimes represented 
by a number or percentage of the results produced — 
for example, the total number of faculty attending the 
training, the percentage of workshops facilitated, or the 
percentage of faculty showing a satisfaction level of 80 
percent or higher on their training. 

Outcomes are the short-, medium-, and long-term effects 
or changes the intervention is designed to bring about. 

For example, these could include changes in participants’ 
learning, actions, skills, or opinions (short-term); changes 
in behavior, social actions, or decision-making (medium- 
term); and changes in economic, social, and environmental 
conditions (long-term). 
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Figure 1 


Typical Elements of a Logic Model 


Input 


Activity 


Output 


Outcome 

Resources needed: humen, ■ 
financidi, organizationdl, etc. 


Tasks that use your resources ■ 
in order to produce cn output 


Results or products generoted ■ 
by the completioh of your 
activities 


Effects or chauges the 
inten/ehtioh makes on 
participants 


When developing a logic model for an intervention, 
consider following the typical “left-to-right” process for 
identifying the inputs, activities, outputs, and outcomes. 
Others prefer to “begin with the end in mind” and work 
backward, that is, they first identify the ultimate goals or 
outcomes the intervention is designed to accomplish and 
then identify the outputs, activities, and necessary inputs. 

Figure 2 shows a logic model in the left-to-right format 
developed by the College of the Mainland, an Achieving 
the Dream institution in Texas. 

Logic models should illustrate the main steps involved in 
implementing interventions. Note that in the logic model 
below. College of the Mainland details the main activities 


that will lead up to its training of faculty and staff. 
Additional steps could be added directly to the evaluation 
plan detailing how the faculty and staff are expected to 
implement this intervention across the college and how 
successful implementation will be ensured. 

When developing logic models, institutions should be 
clear about the assumptions underlying the expected 
outputs and outcomes of the logic model. For example. 
College of the Mainland agreed on the following 
assumptions when developing the logic model for its 
advising system: “The institution ’s current view of the nature 
of advising, which tends to be narrowly focused on the studenfs 
schedule for a particular term, will change dramatically, with 
much greater emphasis placed on placement in the correct 


Figure 2 

College of fhe Mainland’s Logic Model for ifs Revised Advising Sysfem 


Faculty 

Advisors 

Staff 

Registrar 


Activity 


Canvene advising 
task force 

\ 

Compiete additionai 
research on 
advisement process 

■f 

Develop academic 
advising model & 
student advisement 
handbook 

\ 

Train faculty & staff 
on new/revised 
advising system 


Output 


Revised and 
approved academic 
advising model & 
student advisement 
handbook 

Faculty & staff 
training modules on 
new/revised advising 
system in place 


Outcome 



Long-term 

Short-term 

Medium-term 

Increased course 
completion rates 

Increased knowledge 

\ 

► of the academic 

Adequate placement of 
students in courses 

Greater retention of 

advising process & 
Texas Success Initiative 

students 

requirements 


Higher number of 
students earning a 
degree or certificate 


Assumptions: The institution's current view of the nature of advising, which tends to be narrowly focused on the student's schedule for a particular term, will 
change dramatically, with much greater emphasis placed on placement in the correct courses, mentoring, etc. There will be a much-reduced reliance on 
part-time advisors during registration periods. As our culture changes to focus on student success and dato-informed decision-making, our improvements may 
actually be exponential. 
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courses, mentoring, etc. There will be a much-reduced reliance on 
part-time advisors during registration periods. our culture 
changes to focus on student success and data-informed decision- 
making, our improvements may actually be exponential. ” This 
statement reveals the college’s overarching assumptions 
about how the advising model will change and what 
outcomes can be expected. Detailing the implementation 
team’s assumptions about how the intervention will work 
helps to frame the description of the intervention, which 
can be useful in communicating with others about what 
the intervention is trying to accomplish and how it is 
expected to do so. 


Developing a logic model takes time, but once completed 
it provides a solid foundation on which to build an 
effective evaluation. There likely will be several rounds 
of revisions before a team will feel satisfied with its logic 
model. Achieving the Dream coaches and data facilitators 
can be a great help in facilitating this process. 

A logic model should be seen as a picture-in-time of 
what an intervention may look like, as envisioned by the 
members of those who designed it. Logic models might 
change as interventions are implemented. Teams should 
review logic models at least twice a year, and make the 
necessary updates to reflect any changes made in the 
interventions. 


Evaluating Student Success Interventions at College of the Mainland, Texas City, TX 

The College of the Mainland (COM) in Texas City, Texas, a eommunity college of approximately 4,000 
students, joined Achieving the Dream in 2006. After completing an extensive review of longitudinal student 
cohort data complemented by qualitative data analysis, the college identified three student success 
priorities. These priorities included the revision of the academic advising system, the implementation of a 
first-year experience program, and professional development for faculty. The evidence gathered and the 
analyses completed suggested that the college develop interventions that could help dose the gap in 
student achievement in each priority area. 

The data about COM's advising system indicated that the system was not functioning well enough to 
support student success. Essential to student success are enforcement of prerequisites, placement of 
students in the correct courses, and faculty-student engagement. To accomplish these goals, the college's 
Achieving the Dream data team proposed to the core team a complete revision of the advising system, 
shifting from largely professional advisors to faculty, with appropriate training provided to support this 
revised advising model. 

As shown in Figure 2, four activities were implemented to achieve this shift in the advising model: (1 ) 
establishment of an advising task force; (2) additional research on the advisement process; (3) the 
development of an academic advising model and a student advisement handbook; and (4) the 
training of faculty and staff on the new advising system. While these activities are key elements in shifting 
the advising responsibilities to faculty, they are not in themselves student success outcomes, Instead, 
the outcomes are the effects that the college hopes to achieve through the implementation of these 
activities. These outcomes are characterized in the logic model as short-term, medium-term, and long- 
term and include: (1) increased knowledge of the academic advising process and Texas Success Initiative 
requirements (short-term); (2) adequate placement of students in courses (medium-term); and (3) 
increased course completion rates, greater retention of students, and higher numbers of students earning 
a degree or certificate (all long-term outcomes), The assumptions or "theory of change" underlying the 
revision of the advising system are noted on the logic model, 


Dr. Pam Millsap, professor of psychology and co-director of COM's Achieving the Dream initiative, 
developed this brief description, She can be reached at pmillsap@com.edu. 


Evaluating Student Success Interventions 


Achieving the Dream: Community Colleges Count 


step 2 : Identify the evaluation questions 

Achieving the Dream encourages colleges to conduct two 
primary types of evaluations: formative and summative. 
During the start-up or pilot period of an intervention, 
colleges may find it useful to conduct formative 
evaluations. The goal of this work is to provide timely 
feedback to faculty, staff, and administrators on whether 
interventions are being implemented as intended, and 
to identify areas for improvement. Once an institution 
is reasonably assured that a program or policy has been 
implemented as intended, it should conduct a summative 
evaluation-that focuses on providing evidence to assist 
decision makers in determining whether an intervention 
should be continued, expanded, or eliminated. Robert 
Stake distinguished these types of evaluation by saying, 
“When the cook tastes the soup, that’s formative 
evaluation; when the guest tastes it, that’s summative 
evaluation” (cited by Scriven, 1991, p. 19). 

Figure 3 shows the different questions that are asked 
when focusing on the inputs^ activities or processes, and 
outputs of an intervention. 

A logic model can provide a useful way to identify exactly 
where the team wants to focus the evaluation and, 
consequently, develop the evaluation questions. 

To define the evaluation questions, a good place to start 
is to hold a brainstorming session in which those who are 
developing the intervention, along with the ATD team 
members, identify a list of possible questions about the 
inputs, process/outputs, or outcomes of the intervention. 
No judgment is made about the importance of one 
question over another. 

Figure 3 


Examples of questions based on the evaluation focus 


Input 

Process/ 

Output 

Outcome 

What resources are 

is the intervention 

To what extent are 
desired changes 
occurring? And for 
whom? 

needed for starting this 

implemented as 

intervention? 

intended? 

How many facuity or 
staff members wiii we 

Are all participants 
being reached as 

Is the intervention 
making any difference? 

need? 

intended? 

What seems to work? 



Not to work? 


Using the College of the Mainland’s logic model as an 
example, questions that could be asked in a formative 
evaluation include: 

■ Do students, faculty, staff, and advisors understand 
the advisement process.? 

■ What elements of the revised advising process are 
perceived as strengths.? 

■ What materials produced in this new advising process 
proved to be most useful.? 

■ What parts of the new advising process could be 
strengthened.? 

An example of a summative evaluation question is: Did 
the revised advisement process positively affect student 
persistence.? Once the team is satisfied with the initial 
list of questions, it is time to select the most important 
ones. Team members might be asked to rank order the 
questions in the list. A useful tool for helping the team 
reach agreement on the final evaluation questions is 
shown in Figure 4. 

The information provided in Figure 4 will help create a 
priority list of questions for the evaluation. 

Step 3: Complete an evaluation plan 

The most important tool for ensuring a successful 
evaluation is the evaluation plan. A well-developed 
evaluation plan helps clarify what is to be evaluated, sets 
priorities for allocating resources, estimates timelines, and 
specifies roles and responsibilities. 

Achieving the Dream institutions vary in their capacity 
to plan and their skill in executing evaluations of 
their interventions. Colleges develop their evaluation 
plans in consultation with their Achieving the Dream 
coach and data facilitator. Data facilitators in particular 
have extensive experience with the use of evaluation 
methodologies, and are equipped with tools and resources 
to make this process easier. 

Figure 5 shows the elements of an evaluation plan. 
Column A lists the main evaluation questions. Column B 
presents the expected outcomes that were identified in 
the logic model developed for this specific intervention. 
This helps highlight the relationship between the 
evaluation questions and expected outcomes. Column C 
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Figure 4 


Example of information to identify final evaluation questions* 


Evaluation question 

Potential benefits of being 
able to answer this question 

Feasibility of obtaining 
data needed to answer 
this question 

Time and resources required 
to answer this question 

Question 




Question 




Question 





• Determine your evaluation questions in column one ttien move left to right, completing the table to identity the tinol evaluation questions. 


shows the specific tasks that need to be completed to 
answer each evaluation question. Column D is a list of 
the personnel who will be charged with the execution 
of the evaluation task. Column E indicates the timeline 
for executing each task. Column F lists the data sources 
for the information that will help answer each evaluation 
question. 

Column G presents how the data collected will be 
analyzed. It also lists the methods that will be used to 
analyze the data (e.g., descriptive statistics, t-tests, 
one-way ANOVA, etc.). Column H on Reporting 
to how, when, and with whom the evaluation results 
will be shared. 


The data used for an evaluation depend on the questions 
to be answered. Quantitative data normally help 
you answer the “what” of your evaluation questions. 
Qualitative data help you answer the “why” of your 
evaluation questions. 

For example, when developing and implementing a 
new student orientation process, a college may have a 
formative evaluation question, such as: How helpful was the 
advisement training for faculty and staff? \n order to answer 
this type of question, a college might develop a survey for 
faculty and staff who participated in the training to find 
out which parts of the training were most or least useful 
and how they incorporated what they learned into their 
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Figure 5 


College of the Mainland Evaluation Plan 


A. Evaluation 
Questions 

B. Expected 
Outcomes 

C. Tasks 

D. Personnel 

E. Timing 

F. Data Source 

G. Analysis 

H. Reporting 

To what extent 
do faculty, staff, 
and advisors 
understand 
the advisement 
pracess? 

increased knowi- 
edge of academic 
advising process 
and Texas 
Success Initiative 
requirements 

Deveiop survey 
instrument to 
assess the 
effectiveness of 
the training 

Administer survey 
to facuity and staff 

Data team, IR 
staff 

At the end of each 
training session 

Data gathered 
through survey 

Descriptive 

statistics 

ATD core & data 
teams; part- and 
full-time faculty, 
staff, and advisors 

To be shared during 
data & core team 
meetings as results 
become available 

What eiements 
of the revised 
process are 
perceived as 
strengths? 

What needs for 
improvements 
were identified? 

Areas in need of 
improvement wiii 
be addressed 

Deveiop survey 
insfrument(s) 

Administer survey 
to students, 
facuity, and staff 

Deveiop profocoi 
for focus groups 

Train faciiifators 

Conducf focus 
groups wifh faculty, 
staff, and students 

Analyze focus 
group data 

Review results 
with faculty and 
staff, and discuss 
areas in need of 
improvement 

IR staff, data 
team, evaluation 
committee 

Spring 

Data gathered 
through survey 
and focus groups 

Descriptive 
statistics and 
summary of results 
from focus groups 

Core & data teams; 
full- and part-time 
faculty, staff, and 
advisors 

To be shared at 
the end of spring 
ordering summer 
semesters 

To what extent 
did the revised 
advisement 
process posifiveiy 
affect student 
outcomes? 

increased 
successfui course 
compietion rates 

increased retention 
of students 

Higher number 
of degrees and 
certificates 
awarded 

Collect and frock 
data on course 
completions 

Collect and track 
data on student 
retention 

Collect and track 
data on awards 

Data team, 
IR staff 

Fall and winter 
semesters 

Student database 
system 

Longitudinal and 

cross-sectional 

analyses 

Core & data 
teams, faculty, and 
administrators 

To be shared at 
the end of fall and 
winter semesters 
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work with students. This type of evaluation question seeks 
to explain how well an intervention worked and provide 
insight into how it could be further revised or improved. 

A college also might want to ask summative evaluation 
questions, such as: What are the differences in student 
success among subgroups of students who attended the new 
student orientation? question suggests designing an 
evaluation that tracks a cohort of students who received 
an intervention and comparing that to a cohort of students 
who did not. For example, to assess the impact of a 
revamped new student orientation on student success in 
the first term, the college could compare the first-term 
course completion rates for new students who went 
through the new orientation with the course pass rates 
for students from previous entering cohorts with similar 
characteristics (e.g., age, gender, race) who went through 
an earlier version of orientation. This type of evaluation 
method seeks to measure what resulted from the 
intervention. 


Step 4: Monitor execution of the pian 

Once the evaluation plan has been completed, it is time 
to start monitoring its execution. It is recommended that 
one person, perhaps the Achieving the Dream core or 
data team leader, be responsible for monitoring the plan 
and keeping it up to date. For example. College of the 
Mainland’s teams developed a detailed work schedule 
for the evaluation of its interventions (see Figure 6). 

A work schedule is an important tool for monitoring the 
execution of the evaluation. It also informs the decision- 
making processes about the resources being used and 
the potential changes that need to take place during 
the evaluation. 

In a work schedule like the one below, each activity in 
the evaluation plan is broken down into specific tasks. 
Individual faculty and staff members are identified as 
being primarily responsible for the completion of each 
task. In this way, it is easy to monitor the progress of 


Figure 6 

Sample Evaluation Detailed Work Schedule: College of the Mainland 


Year 

Semester 

Target Date 

Task# 

Task 

Dote Team 

Other 

Personnel 

Date 

Completed 

07-08 

Fall 07 

Nov 

1 

Collect and track data on 
student retention F-06-Spr 07 
and F-06-F-07 

IR REP 

IR 

Feb 08 

07-08 

Fall 07 

Dec 

1 

Collect and track data on 2007 
awards 

IR REP 

IR 

Dec 07 

07-08 

Fall 07 


1 .a 

Develop instruments to access the 
effectiveness ot advisement training 

FAC & IR REP 

Advising Task 
Force 

Oct 07 

07-08 

Fall 07 


1 .a 

Administer advisement training 
instrument to faculty and staff 

IR REP 

Advising Task 
Force 

Nov 07 

07-08 

Fall 07 


l.b 

Communicate with Faculty and 
Staff about how to provide their 
feedback 

FAC & IR REP 


Nov 07 

07-08 

Fall 07 


l.b 

After communication with Faculty 
and Staff, an appropriate method 
will be selected to assess strengths 
and weaknesses otthe advising 
process 

FAC & IR REP 


Nov 07 

07-08 

Fall 07 


l.b 

Develop survey instrument to assess 
strengths and weaknesses of revised 
advisement process (if deemed 
necessary) 

FAC & IR REP 

Advising Task 
Force 

NA 
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evaluation activities and hold team members accountable 
for completing the work in a timely fashion. 

It is important to note that neither the logic model nor the 
evaluation plan should be seen as set in stone. Continuous 
reviews and updates are highly encouraged as the 
interventions change direction based on the experience 
of those implementing the intervention and the findings 
from the evaluation itself Institutions are encouraged to 
update their logic models and overall work plans at least 
twice a year. 

Step 5: Learn, share, and use the results 

Team members gather data that can help them learn 
whether an intervention is producing the desired 
outcomes and how its effects can be improved. What 
colleges learn from evaluations can be very informative 
to those involved in implementing interventions as well 
as others who are not directly involved, but are seeking 
ways to improve student success. However, an evaluation 
is only successful if the results are used. Not using the 
results of an evaluation would mean a waste of resources 
(i.e., time, money, material, students’ efforts, completed 
research) invested in conducting it. Even if evaluations do 
not show the expected results, they still can be of value. 
They might provide an opportunity for implementers 
to regroup, revisit, and propose a new course of action, 
including modifying the intervention if necessary, adding 


new data-gathering strategies, or enlisting the help of an 
outside professional evaluator. 

To ensure that the results from evaluations have a 
positive impact on student outcomes, colleges need to 
communicate the findings of evaluations broadly and 
convene groups of faculty, staff, and administrators 
to review the results and discuss the implications for 
improving institutional policy and practice. Ideally, 
there is a give-and-take in which the practitioners 
reflect on how to use the evaluation findings to continue 
improving programs and services, and those in charge 
of the evaluation continue evaluation activities to see 
whether the attempted improvements actually produced 
the desired results. Sharing evaluation results widely 
and encouraging their regular use for improvement are 
key steps in building a “culture of evidence” to increase 
success for college students. 
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a data facilitator for two colleges in Texas and one in 
South Carolina. 
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